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Abstract1 


The  acquisition  and  engineering  of  large  scale,  complex  information  systems,  particularly 
those  that  transcend  organizational  and  functional  boundaries,  represent  well-recognized 
challenges.  The  processes  and  procedures  that  evolved  during  the  second  half  of  the  20th 
century  are  best  suited  for  the  development  of  linear,  well-bounded  systems.  These  processes 
have  proven  difficult  to  adapt  to  situations  in  which  stakeholders  do  not  always  agree, 
requirements  evolve,  and  constraints  keep  changing.  Different  processes  and  procedures  are 
needed  to  address  the  acquisition  and  engineering  of  information  technology  systems  with 
evolving  requirements  and  rapidly  changing  technologies. 

This  paper  is  based  on  the  results  of  a  multi-year  research  program  that  investigated 
how  uncertainty-based  acquisition  methods  can  be  used  to  improve  the  odds  of  successful  IT 
acquisitions.  The  paper  presents  new  concepts  for  managing  uncertainty  in  acquisition 
programs:  the  uncertainty  landscape,  uncertainty-driven  acquisition  strategies  (staged 
commitment,  small  bets),  the  Y  model,  and  a  three-step  approach  to  implementation  (i.e., 
diagnosis,  strategy  selection,  and  adaptive  execution).  More  than  20  acquisition  programs  were 
studied,  and  pilot  programs  were  initiated  to  test  the  frameworks  and  strategies  suggested  in  the 
research. 


I.  Introduction 

Traditional  systems  engineering  and  acquisition  practices  evolved  during  the  second  half 
of  the  20th  century  primarily  to  deal  with  the  particular  challenges  of  developing  large-scale 
weapon  systems.  These  systems  are  expected  to  have  long  service  lives,  often  measured  in 
decades,  and  typically  require  development  and  harnessing  of  unique,  breakthrough 
technologies.  Because  of  these  challenges,  development  occurs  over  a  multi-year  period 
before  these  systems  enter  production.  Changes  late  in  development  or  in  production  directly 
contribute  to  cost  and  schedule  overruns.  Consequently,  once  the  design  is  completed,  there  is 
strong  resistance  to  change,  and  rightfully  so.  The  ideal  situation  for  these  types  of  programs  is 
one  in  which  requirements  remain  relatively  stable  and  critical  technologies  mature  over  the 
course  of  the  development  period  (Stevens,  forthcoming).  Related  to  this,  there  is  growing 
Congressional  emphasis  on  better  up-front  planning  and  governance  practices  that  focus  on 
controlling  deviations  from  the  plan  (Levin  &  McCain,  2009). 

Information  technology  (IT)  systems,  particularly  those  that  provide  user-facing 
applications,  pose  different  challenges.  These  systems  are  often  intended  to  operate  in  highly 
volatile  environments  and,  thus,  are  subject  to  changing  user  needs  and  expectations.  In  the 
most  volatile  environments,  the  effective  life  of  IT  systems  can  be  measured  in  weeks  to  months 
rather  than  years.  Development  and  acquisition  tempos  have  to  be  responsive  to  such  urgent 
and  short-lived  needs.  Further,  these  systems  often  leverage  commercial  technologies  that  are 
also  rapidly  evolving.  Unlike  weapons  systems  in  which  change  is  rightfully  something  to  be 
controlled,  for  many  information  technology  systems,  change  is  inevitable  and  must  be 
accommodated.  For  these  systems,  there  is  a  risk  that  requirements  are  locked  in  too  early  and 
may  not  be  responsive  to  legitimately  changing  user  needs  and  that  technologies  become 
outdated  while  the  system  is  still  in  development. 


1  MITRE  Public  Release:  09-1310 
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Traditional  processes  and  procedures  are  best  suited  for  the  development  of  linear,  well- 
bounded  systems  and  have  proven  difficult  to  adapt  to  situations  in  which  stakeholders  do  not 
always  agree,  requirements  evolve,  and  constraints  keep  changing.  Different  processes  and 
procedures  are  needed  to  address  the  acquisition  and  engineering  of  IT  systems,  particular 
those  with  evolving  requirements  and  rapidly  changing  technologies. 

Current  systems  engineering  and  acquisition  practices  that  are  optimized  to  deal  with  the 
unique  challenges  and  risks  in  the  development  of  weapon  systems  do  not  provide  the  flexibility 
and  agility  needed  to  deal  with  the  uncertainties  inherent  in  many  IT  systems  acquisitions.  A 
tailored  approach  to  IT  acquisition  that  explicitly  acknowledges  the  inherent  uncertainties  and 
provides  the  necessary  flexibility  is  required. 

The  challenge  of  developing  and  acquiring  IT-based  systems  more  rapidly  and  with 
greater  agility  complements  and  does  not  supplant  the  very  real  and  widely  recognized 
challenges  of  developing  and  acquiring  weapon  systems.  In  fact,  Dr.  Ashton  Carter,  recently 
confirmed  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  in  reply  to 
advance  questions  from  the  Senate  Armed  Services  Committee,  pointed  out  these  two 
challenges: 

A  first  major  challenge  is  to  ensure  that  AT&L  is  supporting  the  war  effort  through  rapid 
acquisition  of  systems  our  soldiers,  sailors,  airmen,  and  Marines  need  in  Iraq, 
Afghanistan,  and  in  the  war  on  terror ....  A  second  major  challenge  is  to  get  under 
control  the  many  troubled  acquisition  programs  that  are  supposed  to  be  supporting  our 
forces — both  today  and  tomorrow.  Too  many  of  these  programs  are  failing  to  meet  their 
cost,  schedule,  and  performance  expectations,  and  some  are  failing  even  more 
fundamentally  the  test  of  whether  they  are  needed  for  the  future  military  challenges  we 
are  most  likely  to  face.  In  addition  to  disciplining  these  programs,  reform  of  the 
acquisition  system  is  needed  to  ensure  that  we  do  not  get  ourselves  in  this  position 
again  in  the  future.  (Carter,  2009) 

This  paper  reports  on  research  conducted  by  The  MITRE  Corporation  to  examine 
alternative  acquisition  strategies  and  practices  for  IT  systems  under  varying  conditions  of 
uncertainty.  Alternative  strategies,  tailored  to  an  understanding  of  the  nature  and  extent  of 
uncertainty  faced  by  the  program,  are  proposed. 

The  paper  is  organized  as  follows: 

■  Section  I  introduces  the  need  for  new  acquisition  approaches  for  uncertainty. 

■  Section  II  discusses  the  regulations  that  permit  flexible  acquisition  approaches. 

■  Section  III  describes  the  three  phases  of  this  research  program. 

■  Section  IV  highlights  the  key  research  findings,  including  an  uncertainty  landscape 
and  strategies  for  dealing  with  uncertainty. 

■  Section  V  describes  a  three-step  approach  for  implementing  the  strategies. 

■  Section  VI  introduces  a  model  to  describe  implementation  drivers,  enablers,  and 
barriers.  Section  VII  summarizes  the  acquisition  and  systems  engineering 
implications  of  this  research. 
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II.  Regulations  Support  Tailoring  Acquisition  Practices 

Language  in  federal  acquisition  regulations  specifically  encourages  the  acquisition  team 
to  institute  innovative  practices  tailored  to  the  particular  needs  and  circumstances  of  the 
program.  In  particular,  the  Federal  Acquisition  Regulation  (FAR)  Part  1.102:  Statement  of 
Guiding  Principles  for  the  Federal  Acquisition  System  states: 

(d)  The  role  of  each  member  of  the  Acquisition  Team  is  to  exercise  personal  initiative 
and  sound  business  judgment  in  providing  the  best  value  product  or  service  to  meet  the 
customer’s  needs.  In  exercising  initiative,  Government  members  of  the  Acquisition  Team 
may  assume  if  a  specific  strategy,  practice,  policy  or  procedure  is  in  the  best  interests  of 
the  Government  and  is  not  addressed  in  the  FAR,  nor  prohibited  by  law  (statute  or  case 
law),  Executive  order  or  other  regulation,  that  the  strategy,  practice,  policy  or  procedure 
is  a  permissible  exercise  of  authority.  (GSA,  2005) 

Similarly,  DoD  Instruction  5000.02  Section  1,  Defense  Acquisition  Management  System 
(USD  (AT&L),  2008)  states: 

b.  Consistent  with  this  Instruction  and  Reference  (b),  the  Program  Manager  (PM)  and 
the  MDA  shall  exercise  discretion  and  prudent  business  judgment  to  structure  a  tailored, 
responsive,  and  innovative  program. 

Despite  regulations  that  allow  flexible  acquisition  methods,  there  is  little  guidance  on 
what  these  methods  should  be.  This  research  was  undertaken  to  identify  particular  methods. 

III.  Research  Approach 

The  research  explores  strategies  and  methods  for  managing  uncertainty  and  offers 
opportunities  to  address  federal  IT  acquisition  challenges.  The  objectives  of  the  research  are  to 
determine:  (1)  how  best  commercial  practices  for  dealing  with  uncertainty  can  be  adapted  to 
federal  IT  acquisition,  (2)  under  what  circumstances  might  they  work  and  make  a  difference, 
and  (3)  what  needs  to  change  to  make  it  happen.  Research  activities  are  organized  into  three 
phases  as  illustrated  in  Figure  1 . 
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Phase  I:  Literature  Review, 

Analysis,  and  Concept 

Development 


Phase  II:  Field  Research  and 

Development  of  Uncertainty 

Based  Strategies 


Phase  III:  Toolkitfor  Program 

Managers  and  Acquisition 

Professionals 


Acquisition  Processes  and 
Regulations 

FAR.  DFARS, 

DoD  5000 


Commercial  Practices 


Selection 

•Market  Focus 
•Syndication 
•Due  diligence 

Diversification 

•Portfolio  of  many  small 
bets 

•Rapid  start  and  stop 

Agility 

•Funding  tied  to  achieving 
“inch  stones” 

•Flexible  financing 
•Speed  to  market  with 
feedback 

Active 

Involvement 

•Strategic  advice 
•Teambuilding 
•Leveraging  contacts 

Case  Notes  and  Pilot  Study 


i 


Uncertainty  Based  Strategy  Patterns 
and  Enabling  Practices 


Uncertainty 
Diagnostic  Tool 


/ 


/ 


“How  To...”  Guide 


Figure  1.  Three  Research  Phases 

Phase  I 

Phase  I  of  the  research  characterized  and  compared  commercial  entrepreneurial 
(McGrath  &  MacMillan,  2000)  and  federal  acquisition  environments  and  methods  to  develop  a 
research  framework  to  assess  the  nature  and  level  of  uncertainty  within  an  acquisition  program. 
The  research  was  based  on  literature  reviews  and  analyses  of  federal  acquisition  processes 
and  regulations,  as  well  as  interviews  with  venture  capitalists,  entrepreneurs,  government 
program  and  project  managers,  and  acquisition,  contract,  and  budget  specialists.  The  concepts 
and  frameworks  were  refined  based  on  discussions  with  commercial  entrepreneurs,  government 
specialists,  and  academicians. 

Phase  II 

Phase  II  developed  and  detailed  alternative  acquisition  strategies  for  dealing  with 
uncertainty  and  initiated  field  research.  The  field  research  was  an  embedded  multiple-case 
study  (Yin,  2002)  and  included  data  from  more  than  20  programs  from  12  government  agencies 
responsible  for  the  acquisition  of  information  intensive  systems.  Working  from  an  interview 
protocol,  the  research  team  collected  information  on  the  programs’  acquisition,  development, 
contract,  incentive,  and  governance  strategies.  The  case  notes  provided  input  for  identifying 
acquisition  strategies  for  dealing  with  uncertainty  and  for  exploring  the  motivators,  enablers,  and 
barriers  to  innovation  in  acquisition  practice.  A  pilot  study  with  an  active  acquisition  program 
was  initiated  in  Phase  II  to  test  and  validate  the  strategies  proposed  in  the  research. 

Phase  III 

Phase  III  focused  on  synthesizing,  validating,  and  communicating  the  research. 

Research  findings  were  captured  in  an  interactive  diagnostic  tool  and  a  "How  to  Guide"  for 
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program  and  project  managers  and  key  members  of  the  acquisition  team.  A  model  for 
assessing  an  organization’s  readiness  to  implement  uncertainty-based  strategies  was  then 
developed. 

IV.  Research  Findings 

Acquisition  Uncertainty  Landscape 

IT  acquisitions  face  both  internal  and  external  uncertainty.  Within  a  typical  acquisition, 
the  team  faces  internal  uncertainties  in  design,  implementation,  and  performance.  Even  though 
the  members  of  the  team  have  had  similar  experiences,  there  are  still  many  new  aspects  to 
address.  The  team  can  be  uncertain  about  how  to  design  the  system,  how  to  optimize  the 
implementation,  and  how  well  the  system  will  meet  functional  and  performance  requirements. 

A  typical  acquisition  also  encounters  even  more  challenging  external  uncertainties. 
These  include  changes  in  the: 

■  Operational  environment,  threat  or  mission, 

■  Business  processes,  governing  policies  and  regulations, 

■  User  requirements  and  expectations, 

■  Priorities, 

■  Competitors  (including  user-initiated  efforts), 

■  Technologies,  and 

■  Stakeholder  actions  and  influence. 

Internal  and  external  factors  translate  into  two  critical  dimensions  of  uncertainty: 
uncertainty  about  what  to  build  and  uncertainty  about  how  to  build  it.  Figure  2  describes  an 
acquisition  uncertainty  landscape  based  on  two  key  dimensions  of  uncertainty:  evolving 
requirements  and  emerging  technologies.  In  the  lower  left  hand  corner  of  the  figure, 
requirements  are  relatively  stable,  and  technologies  are  mature.  This  indicates  less  uncertainty 
and,  therefore,  more  predictability  in  the  execution  and  outcome  of  the  program. 
Program/project  managers  know  more  clearly  what  needs  to  be  built  and  the  appropriate 
approach  to  building  it.  A  traditional  acquisition  approach  often  works  well  in  this  predictable 
area  of  the  landscape. 


DEFENSE  ACQUISITION  IN  TRANSITION 


-79- 


Uncertainty  about  what  to  build 


<D 

V) 

c 

_Q 

CO 

<D 

E 

<D 

CO 

i— 

<1) 

'd 

i— 

cr 

O 

<D 

2 

More 

Prei 

dictabilit] 

/ 

More  Mature 
Technologies 


New  or  Emerging 
Technologies 


Figure  2.  Acquisition  Uncertainty  Landscape 

Farther  away  from  the  lower  left  hand  corner,  evolving  requirements  and  emerging 
technologies  introduce  more  uncertainty.  The  traditional  approach  of  locking  down  requirements 
early  and  following  a  waterfall  development  effort  do  not  work  as  well.  Acquisition  strategies 
must  expect  and  accommodate  change  and  build  in  the  flexibility  for  dealing  with  uncertainty. 

A  basic  premise  of  this  research  is  that  IT  acquisition  must  take  into  account  a  program’s 
position  on  the  uncertainty  landscape.  Acquisition  strategies  should  be  selected  depending  on 
the  nature  and  scope  of  the  underlying  uncertainties.  In  addition,  not  all  parts  of  a  program 
demonstrate  the  same  type  and  degree  of  uncertainty.  For  example,  development  of  the  basic 
infrastructure  may  be  more  predictable  and  fall  in  the  lower  left  hand  quadrant  of  the  landscape, 
while  user  facing  services  are  often  closer  to  the  upper  right  hand  quadrant  due  to  changing 
user  expectations  and  new  technologies. 

As  shown  in  Figure  3,  the  various  components  (“chunks”)  of  a  program  may  belong  in 
different  locations  on  the  uncertainty  landscape.  Therefore,  not  all  parts  of  a  program  have  the 
same  need  for  flexibility.  It  is  useful  to  envision  the  components  of  a  program  as  constituting  a 
portfolio.  The  core  elements  of  the  portfolio  are  those  that  are  more  predictable  and  can  be 
managed  using  a  classic  approach,  while  the  more  uncertain  components  require  a  more 
flexible  strategy. 
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Figure  3.  Mapping  Program  Components  on  the  Acquisition  Uncertainty  Landscape 
Strategies  for  Dealing  with  Uncertainty 

This  research  found  that  different  strategies  are  appropriate  for  different  levels  and  types 
of  uncertainty.  A  plan-driven  strategy  works  best  when  requirements  are  primarily  stable  and 
technologies  are  mature.  As  illustrated  in  Figure  4,  staged  commitment  and  small  bets 
strategies  are  more  appropriate  for  components  facing  uncertainty  about  what  to  build  and  how 
to  build  it,  respectively.  These  findings  are  consistent  with  earlier  research  in  project 
management  (Loch,  DeMeyer  &  Pich,  2006),  product  development  (Smith,  2007)  and  venture 
capital  (Gompers  &  Lerner,  2001:  Stevens,  King  &  Halley,  2008). 
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Figure  4.  Strategies  Used  to  Manage  Uncertainty 

The  following  paragraphs  outline  three  strategies  for  different  areas  of  the  Uncertainty 
Landscape. 

Strategy  1:  Plan-driven 

When  there  is  little  uncertainty  in  requirements  and  technology,  a  program/project 
manager  can  successfully  use  the  traditional  Plan-Driven  Strategy.  This  traditional  method 
consists  of  defining  a  set  of  requirements,  design,  cost,  and  schedule,  and  then  carrying  out  the 
associated  plan.  Since  there  should  be  little  need  to  change,  progress  is  measured  against  the 
plan,  and  success  is  determined  by  how  closely  cost,  schedule,  and  requirements  goals  are 
met.  Project  management  techniques,  such  as  Earned  Value  Management  (EVM),  have  been 
developed  to  measure  execution  against  the  plan.  This  plan-driven  method  has  been 
successfully  used  for  years  when  there  is  little  doubt  about  what  is  needed  and  how  to  satisfy  it. 
Where  risk  is  present,  program/project  managers  use  well-recognized  risk  management 
techniques.  The  idea  is  to  “make  a  plan  and  execute  to  the  plan.” 

Strategy  2:  Staged  Commitment 

When  there  are  rapidly  evolving  requirements  and  uncertainty  about  what  to  build,  a 
program/project  manager  can  follow  an  iterative,  learning  strategy  and  manage  the  project  by 
staged  commitment.  In  a  staged  commitment  approach,  the  acquisition  is  structured  so  that 
funding  and  payment  decisions  are  made  based  on  small  increments  of  demonstrable 
capability.  Staged  commitment  enables  the  program/project  manager  to  scan  the  environment, 
assess  uncertainty  at  each  stage,  and  adjust  the  direction  appropriately.  The  approach 
preserves  the  option  to  re-baseline,  re-direct,  or  terminate  the  effort  if  market  conditions  have 
changed,  priorities  have  shifted,  or  the  contractor  is  not  delivering  as  expected. 
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A  staged  commitment  strategy  is  warranted  when  there  are  uncertainties  in  the  mission, 
business  environment,  or  user  expectations  about  particular  features  or  performance.  In 
addition,  a  staged  commitment  strategy  is  desirable  when  there  are  uncertainties  about  a 
provider’s  ability  to  achieve  agreed-to  objectives.  Deliverables  at  each  stage  are  concrete  and 
can  involve  either  early  prototypes  or  small  increments  of  delivered  capability.  They  can  be 
periodically  evaluated  using  feedback  from  intended  users  and  other  critical  stakeholders.  From 
a  contracting  standpoint,  staged  commitment  can  be  accomplished  through  such  mechanisms 
as  shorter  duration  contracts/task  orders,  with  renewal  options. 

Strategy  3:  Small  Bets 

When  there  are  new  or  emerging  technologies,  program/project  managers  can  apply  a 
small  bets  strategy.  In  a  small  bets  strategy,  parallel  efforts  are  initiated  to  determine  technical 
solutions  and  a  way  forward;  the  pharmaceutical  industry  makes  particular  use  of  this  method 
(Eliasberg  &  Ding,  2002).  Simultaneous,  parallel  mini-projects  are  initiated,  each  one  with  a 
different  design  approach.  The  projects  are  completed,  and  the  design  with  the  most  successful 
outcome  is  adopted.  In  this  way,  many  design  alternatives  can  be  assessed  by  field  trial  and 
experimentation. 

“Small  bets”  strategies  are  used  to: 

■  Explore  alternative  designs  and  approaches  to  implementation 

■  Assess  alternative  technologies 

■  Initiate  one  or  more  parallel  activities  for  high-risk  components  on  the  critical  path 
with  the  understanding  that  there  will  be  a  down-select  decision2 

■  Foster  competition 

■  Hedge  against  the  risk  of  failure  of  one  contractor/provider  or  solution 

An  IT  program  using  small  bets  should  be  one  that  can  be  structured  into  small, 
concurrent  increments  that  can  each  be  developed  and  acquired  independently  and 
subsequently  integrated.  One  critical  method  to  enable  a  small  bets  approach  is  the  creation  of 
contract  flexibility,  which  supports  key  decisions  to  modify  the  acquisition  strategy  efficiently  and 
with  controlled  impact  to  cost  and  schedule. 

Figure  5  illustrates  sample  strategies  and  compares  the  more  traditional  approaches  of 
“one  big  bet”  and  “one  bet  with  learning”  with  variants  of  “small  bets.”  The  small  bets  strategies 
included  in  Figure  5  are: 

■  Design  fly-off:  Multiple  designs  are  initiated  with  a  competition  to  select  the  best 
design.  The  winning  design  is  then  carried  through  to  the  end  of  the  project. 

■  DARPA  model:  Multiple  prototypes  are  carried  forward  in  parallel  through  design, 
implementation,  and  field  test.  Only  the  winner  is  carried  into  operations. 

■  Two-button  model:  Multiple  systems  are  presented  to  the  user.  The  marketplace 
then  decides  which  system  best  fits  their  needs,  rather  than  the  development  team. 


2  This  is  often  initiated  during  design  or  early  prototyping  but  can  also  be  used  downstream  if  the  program 
encounters  design  or  contractor/solution  issues. 
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Figure  5.  Variants  of  Small  Bets 

When  both  requirements  and  technology  uncertainties  exist,  the  program/project 
manager  can  use  a  combination  of  staged  commitment  and  small  bets  strategies.  Under  these 
circumstances,  the  program/project  manager  could  incrementally  commit  to  requirements,  and 
within  each  increment,  initiate  alternative  designs  to  reduce  technology  risk. 

V.  Implementation:  A  Three-step  Approach 

Programs  that  are  the  most  likely  candidates  for  implementation  of  the  alternative 
practices  introduced  above  are  those  that: 

■  Are  initiating  a  novel  endeavor  that  demands  a  different  approach, 

■  Have  attempted  to  use  traditional  strategies,  sometimes  more  than  once,  and  found 
that  they  proved  to  be  unsuccessful,  and 

■  Anticipate  considerable  uncertainty  and  volatility  either  in  terms  of  requirements 
and/or  enabling  technologies. 

These  circumstances  provide  the  necessary  motivation  to  try  alternative  strategies.  In 
addition,  the  program/project  manager  must  be  willing  to  try  these  non-traditional  practices,  and 
the  organization  must  be  willing  to  support  the  effort.  In  effect,  senior  management  must 
provide  the  necessary  encouragement  and  “top  cover.”  Finally,  the  program  must  lend  itself  to 
being  structured  into  small  increments  that  can  be  developed  and  acquired  independently  and 
subsequently  integrated  to  yield  the  desired  capability. 

The  implementation  approach  is  organized  into  three  steps  (Figure  6). 
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Figure  6.  Three-step  Approach  to  Implementation 

Step  1:  Diagnosis 

The  diagnosis  step  is  used  to  determine  which  parts  of  the  program  are  subject  to 
uncertainty,  as  well  as  the  nature  and  magnitude  of  the  uncertainty.  The  underlying  premise  is 
that  uncertainty  is  not  distributed  evenly.  Different  parts  of  the  program  face  different  sources 
and  levels  of  uncertainty.  Examples  of  uncertainties  that  may  be  faced  include: 

■  Changes  in  the  mission  environment  or  underlying  business  process  that  result  in 
changing  user  needs  and/or  expectations, 

■  Uncertainties  about  user  expectations  and  user  acceptance  of  particular  features 
and/or  levels  of  performance, 

■  The  pace  of  change  of  the  enabling  technologies, 

■  Uncertainties  about  the  ability  of  providers  to  achieve  agreed-to  objectives  within  the 
time  and  resource  constraints, 

■  Items  that  are  on  the  critical  path  for  which  risk-hedging  strategies  are  warranted, 
and 

■  Desire  to  foster  competition  of  ideas,  approaches  and/or  technologies. 

The  diagnosis  step  maps  each  of  the  relevant  project  components  onto  the  uncertainty 
landscape  and  suggests  strategies  that  may  be  suitable  based  on  the  nature  and  degree  of 
uncertainty.  The  research  effort  has  developed  an  interactive  tool  to  assist  in  conducting  the 
diagnosis.  Figure  7  provides  an  example  output  of  the  diagnosis  step  for  one  of  the  case 
studies  in  this  research.  (Note:  the  project  name  is  disguised  for  anonymity). 
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Figure  7.  Example  Diagnostic  Output 


Step  2:  Select  and  Implement  the  Acquisition  Strategy 

In  this  step,  the  program/project  manager  plans  for  and  implements  the  selected 
strategies  to  provide  a  set  of  options  that  can  be  exercised  as  situations  warrant.  In  order  to 
preserve  the  flexibility  to  respond  to  potential  uncertainties,  conditions  have  to  be  built  in  early  in 
the  planning  phases  of  the  project.  These  conditions  address  program  structure,  funding, 
contracting,  and  contract/contractor  performance.  In  both  the  staged  commitment  and  the  small 
bets  strategies,  the  project  uses  a  modular  system  design  and  lends  itself  to  modular 
development.  An  overarching  architecture  is  critical  to  defining  the  components,  their 
relationships,  and  interactions  enabling  their  integration. 

The  enabling  conditions  for  the  staged  commitment  strategy  include: 

■  Development  approach:  Adopt  an  agile  or  spiral  development  approach  to  delivery 
of  capability. 

■  Funding:  Align  funding  with  the  system  architecture  and  project  structure.  Arrange 
for  payments  to  be  made  based  on  achieving  user  needs  in  increments  of  capability. 

■  Contracting:  Structure  contract  vehicles  to  allow  for  incremental  periods  of 
performance  (modular  contracting),  including  the  options  for  rapid  termination,  if  and 
when  needed. 

■  Contract/contractor  performance:  Emphasize  outcomes  (capability  delivered). 

■  Incentives:  Structure  contractor  award  to  delivery  of  capability — reflecting  user 
feedback. 

The  enabling  conditions  for  the  small  bets  strategy  include: 
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■  Development  approach:  Adopt  a  modular-based  acquisition  strategy. 

■  Funding:  Provide  fiscal  flexibility  to  support  establishments  of  parallel  activities  as 
needed. 

■  Contracting:  Structure  contract  vehicles  to  allow  for  competitive  parallel 
developments. 

■  Contract/contractor  performance:  Define  business  model  to  foster  desired 
contractor  behavior. 

■  Incentives:  Structure  contractor  award  to  delivery  of  capability — reflecting  user 
feedback. 

These  strategies  place  greater  emphasis  on  initial  and  recurring  trade-space 
assessment  that  address  not  only  cost,  schedule  and  performance  but  also  affordability, 
responsiveness  to  changing  needs,  capability  delivered,  and  interoperability  with  other  internal 
components  and  external  systems.  For  such  strategies  to  be  effective,  and  not  overly  costly, 
there  is  a  need  to: 

■  Limit  documentation  to  what  is  necessary, 

■  Find  more  cost  efficient  ways  to  evaluate  working  prototypes  and  early  capability 
increments, 

■  Implement  a  “light-weight”  governance  process  for  small  increments  and  a  more 
traditional  governance  process  for  major  milestones,  and 

■  Encourage  competition  to  get  better  products,  sooner,  at  lower  cost. 

Step  3.  Adaptive  Execution 

The  program/project  manager  who  must  operate  in  the  today’s  uncertain  environment 
requires  a  degree  of  flexibility  to  make  informed  decisions  and  to  adapt  to  changing 
circumstances.  Laying  the  groundwork  in  the  acquisition  strategy  is  critical.  So,  too,  is  the  ability 
to  continuously  monitor  the  situation  to  identify  changes  in  demand,  priorities,  the  delivery  of 
products,  and  the  “fit”  between  the  product  delivered  and  user  expectations.  If  the 
program/project  manager  views  and  manages  the  program  as  a  portfolio  of  interconnecting 
components,  he  or  she  is  prepared  to  rebalance  the  portfolio  as  warranted.  This  includes 
redirecting,  accelerating,  slowing,  or  terminating  future  increments  and  initiating  risk-hedging 
initiatives. 

VI.  Drivers,  Enablers,  and  Barriers  to  Adopting  Uncertainty- 
based  Strategies 

Although  the  strategies  proposed  in  this  paper  for  addressing  uncertainties  in  IT 
acquisition  are  supported  and  encouraged  by  statute  and  regulations,  in  some  instances  these 
strategies  may  be  viewed  as  non-traditional  and  meet  with  resistance.  To  assess  an 
organization’s  readiness  to  implement  uncertainty-based  strategies,  the  diagnostic  tool  includes 
a  framework  for  program/project  managers  and  acquisition  professionals  to  assess  whether 
there  is  “fertile  ground”  for  implementing  the  proposed  strategies.  Will  the  strategies  be 
successfully  adopted  or  challenged  by  long-standing  practices  and  culture? 

Throughout  the  field  research,  the  team  observed  several  motivators,  enablers,  and 
barriers  to  trying  a  new  acquisition  approach.  The  findings  in  the  case  notes  and  the  extensive 
literature  on  innovation  provide  the  basis  for  a  model  of  innovation  readiness.  The  “Y”  Model 
(named  because  of  its  shape)  is  illustrated  in  Figure  8. 
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Figure  8.  "Y"  Model  of  Innovation  Readiness 

(Stevens,  King,  Beard  &  Halley,  2009) 


The  three  dimensions  of  the  “Y”  model  portray  a  range  of  individual  behavioral 
characteristics,  organizational/cultural  characteristics,  and  project  circumstances.  A  program  is 
plotted  on  the  model,  and  the  resulting  triangular  shape  suggests  the  degree  of  alignment 
between  the  need  to  implement  alternative  practices  and  the  ability  and  willingness  of  the 
organization  to  do  so. 

The  project  circumstances  dimension  of  the  model  was  developed  primarily  from 
information  gathered  from  the  case  notes  research.  The  project  circumstances  are  often  the 
motivation  for  trying  an  innovative  acquisition  approach.  Project  circumstances  that  may  serve 
as  drivers  for  innovation  include: 

■  The  program  is  trying  something  novel  that  necessitates  an  innovative  acquisition 
strategy. 

■  The  program  is  responding  to  a  critical  or  urgent  need. 

■  The  program  is  dealing  with  rapidly  changing  circumstances  or  threats. 

■  The  program  has  run  into  difficulties  before  (in  some  cases,  two  or  three  times)  and 
wants  to  try  a  different  approach. 

The  behavioral  characteristics  and  organizational/cultural  characteristics  dimensions  of 
the  model  were  developed  primarily  from  the  literature  on  innovation  (Rogers,  2003)  and 
organizational  change  (Kotter,  2007;  Schein,  2004;  Holt,  Armenakis,  Harris  &  Field,  2007).  The 
individual  behavior  and  organizational  dimensions  of  the  model  form  the  upper  arms  of  the  Y. 
Points  that  are  farther  out  on  the  vectors  represent  a  greater  willingness  to  accept  innovative 
acquisition  approaches. 
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A  mapping  of  the  programs  studied  in  the  field  research  on  the  "Y”  model  highlights 
some  interesting  patterns.  Figure  9  illustrates  a  project  that  maps  to  the  outer  points  on  each 
vector.  In  this  instance,  the  project  was  developing  a  novel  capability — the  project  leader  was 
comfortable  with  uncertainty  and  willing  to  accept  risk,  and  the  senior  leadership  of  the 
organization  was  supportive  and  willing  to  provide  “top  cover.”  This  is  an  example  of  an  aligned 
project. 
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Project 
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Figure  9.  Examples  of  Aligned  Project 

(Stevens,  King,  Beard  &  Halley,  2009) 

Figure  10  provides  an  example  of  a  misaligned  project.  In  this  case,  the  project  was 
developing  a  capability  that  was  fundamentally  new  to  that  agency,  and  the  project  leader  was 
interested  in  implementing  acquisition  strategies  that  were  considered  to  be  innovative  in  that 
environment.  However,  higher-level  management  in  that  agency  actively  discouraged  change. 
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Behavioral 

Characteristics 


Organizational, 

Cultural 

Characteristics 
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Willingto  accept 
risk,  comfortable  with 
u  n  certainty 

Willingto  adoptn 
ideas/practice 

Skeptical  and  cauti 
about  change 

C  om  m  itted  to  existing  proce 


□  isco  u  ra  ges  ch  a  nge 


Values,  encou rages 
innovation 


Encou  rages  flexibility  to  deal 
w  ith  u  nusu al  ci  rcu  mstan  ces 

Rewards  consistency  with  existing 
processes 


'Stable requirements  and 
technologies 

Unwilling  to  repeat  history  of  failed 
proj  ects  -  Jbu  m  e  d  b  ef  oren 

C han  ging  requ  irem  ents  and 
e  m  ergi  n  g  tech  n  olo  gies 

Novel  projector  urgent  need 


Project 

Circumstances 


Figure  10.  Example  of  a  Misaligned  Project 

Plotting  the  overall  program  and  components  of  the  program  on  the  “Y”  model  can 
provide  an  early  visual  indicator  of  potential  roadblocks  to  uncertainty-based  strategies.  A 
misaligned  mapping  helps  focus  attention  on  the  areas  that  need  to  be  addressed  up  front  when 
implementing  innovative  strategies.  The  areas  of  concern  can  become  part  of  a  risk-mitigation 
plan. 


A  word  of  caution  is  warranted.  While  this  model  arose  from  field  research  observations 
and  is  supported  by  a  rich  body  of  literature  (Holt  et  al.,  2007),  it  is  still  a  work  in  progress. 

Much  of  the  literature  focuses  on  the  commercial  and  education  sectors,  not  on  acquisition 
practices  in  the  federal  government.  Follow-on  research  will  be  required  to  refine  and  validate 
the  model  and  extend  its  granularity. 

VII.  Conclusion 

The  research  effort  is  an  ongoing  study  of  acquisition  strategies  suitable  for  dealing  with 
uncertainty,  particularly  as  it  applies  to  IT  acquisitions  in  the  federal  government.  The  primary 
products  of  the  research  are  an  interactive  diagnostic  tool  as  well  as  a  How  to  Guide  that 
provides  actionable  recommendations  for  program  and  project  managers  and  acquisition 
professionals  that  wish  to  implement  these  strategies. 

Key  to  the  successful  implementation  of  these  strategies  is  a  perspective  that  allows  the 
program/project  manager  to: 

■  Understand  that  different  parts  of  the  program  face  different  types  and  degrees  of 
uncertainty  and  urgency, 
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■  Be  prepared  to  tailor  acquisition  strategies,  picking  a  staged  commitment  strategy  in 
which  there  are  uncertain  or  ambiguous  requirements  and  a  small  bets  strategy  to 
hedge  against  risk  or  foster  competition, 

■  Balance  the  need  for  agility  and  discipline,  applying  discipline  in  executing  the 
current  increment  while  being  agile  and  adaptive  in  adjusting  subsequent  increments 
to  changing  needs,  priorities  and  evolving  technologies, 

■  Recognize  that  while  statute  and  regulations  encourage  flexibility  and  innovation, 
non-traditional  acquisition  approaches  are  often  challenged  by  long-standing  practice 
and  culture, 

■  Seek  an  active  partnership  with  the  end  user  and  “top  cover”  within  the  organization, 
and 

■  Strive  to  implement  a  culture  that  emphasized  outcomes,  “time  to  value”  and 
responsiveness. 
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Two  Types  of  Problems 


Tame  Problems 


l — =) 


•  Well-bounded  problems 

•  You  know  when  you’ve  solved 
one 

•  Lend  themselves  to  linear, 
structured  approach 


Wicked  Problems 


•  Messy  problems:  complexity 
uncertainty,  interdependence, 
diverse  interests... 

•  Problems  keep  coming  back 

•  Iterative,  collaborative  approach 


Rittel,  Horst,  and  Melvin  Webber;  1973.  "Dilemmas  in  a  General 
Theory  of  Planning, “  Policy  Sciences,  Vol.  4:  155-169 
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Federal  Acquisition  Challenge 


■  It  takes  too  long,  costs  too  much...  to  sometimes 
buy  the  wrong  thing 

-  Lengthy  history  of  reform 

-  Values:  requirements  stability,  compliance,  control  and 
oversight 

->  Discipline 

■  IT  acquisitions  pose  additional  challenges 

-  Uncertainty  in  requirements,  technology  pace  of  change 
Values:  rapid  response  to  changing  user  needs/external 


events 
->  Agility 
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Research  Objective 


■  Explore  the  hypothesis  that  commercial  practices 
that  work  in  managing  uncertainty  offer 
opportunities  to  address  federal  IT  acquisition 
challenges 

■  Research  Questions: 

-  How  can  commercial  practices  be  adapted  to  federal  IT 
acquisition? 

-  Under  what  circumstances  might  they  work  and  make  a 
difference? 

-  What  needs  to  change  to  make  it  happen? 
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Research  Approach 


Phase  I:  Literature  Review, 

Analysis,  and  Concept 

Development 


Phase  II;  Field  Research  and 

Development  of  Uncertainty 

Based  Strategies 


Phase  III:  Toolkit  for  Program 

Managers  and  Acquisition 

Professionals 


Acquisition  Processes  and 
Regulations 


FAR.  DFARS, 
DoD  5000 


Commercial  Practices 


Selection 

►Market  Focus 
►Syndication 
►Due  diligence 

Diversification 

►Portfolio  of  many  small 
bets 

►Rapid  startand  stop 

Agility 

►Funding  tied  to  achieving 
i:inch  stones"1 
►Flexible  financing 
►Speed  to  market  with 
feedback 

Active 

involvement 

►Strategic  advice 

►Teambuilding 

►Le  ve  ragi  ng  contacts 
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Uncertainty  Based  Strategy  Patterns 
and  Enabling  Practices 


Uncertainty 
Diagnostic  Tool 
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Acquisition  Strategies  Mapped  to  Acquisition 
Uncertainty  Framework 
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How  To  Implement 


Diagnosis  Acquisition  Strategy 


Identify 

1  y 

Develop 

U/ 

Align  Strategy 

Project 

Acquisition 

Components 

Uncertainties 

Strategy 

♦  External 

♦  One  step 

♦  Program 

•  Internal 

s 

♦  Staged 

s 

structure 

Commitment 

♦  Funding 

♦  Small  Bets 

♦  Contracts 

♦  Hybrid 

♦  Incentives 

•  PMO 

Should  you  consider 
alternative  acquisition 
strategies?  If  so,  for 
what  part(s)  of  the 
program? 


What  should  you  do  to  build  in 
the  necessary  acquisition 
flexibility? 
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Adaptive  Execution 


Monitor 

Situation 


Exercise 

Options 


.  External  *  Continue 

♦  Internal  *  Adjust 

♦  Terminate 


What  factors  should  you  consider 
in  deciding  whether  and  when  to 
exercise  flexibility  options? 
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“Y”  Model:  Factors  Impacting  Likelihood  of 
Implementing  Alternative  Practices 


■  Drivers  and  motivators 

Project  characteristics  (inc. 
degree  of  novelty,  experience 
with  failed  projects) 

■  Enablers  and  barriers 

Behavioral  characteristics 
(tolerance  for 
uncertainty/ambiguity. . .) 

Organizational/cultural 
characteristics  (inc.  top  cover) 

■  Environment 

Formal  policy,  statutes 
Technology  pace  of  change... 


Behavioral 

Characteristics 


Willing  to  accept 
risk,  comfortable  with 
u  ncerta inty 


Willing  to  adopt  new 
ideas/practice 


Skeptical  and  cautious 
about  change 


Comm  itted  to  existing  processes 


Organizational, 

Cultural 

Characteristics 


Values,  encourages 
innovation 


Encou  rages  flexibility  to  deal 
with  u  nusu  a  I  circu  mstances 


Rewards  consistency  with  existing 
processes 


Discourages  change 


Sta ble requ irem ents  and 
technologies 

Unwillingto  repeat  history  of  failed 
projects  -  “tau med  bef  ore” 

Chan  ging  requ  irem  ents  and 
e  m  ergi  n  g  tech  n  olo  gies 

i*  Novel  projector  urgent  need 


Project 

Circumstances 


©2009  IEEE 


MITRE 


©  2009  The  MITRE  Corporation.  All  rights  reserved. 


Environment 


Three  Examples  from  our  Case  Notes 


Aligned 


Behavioral 

Characteristics 


Willing  to  accept 
risk,  comfortable  with 
uncertainty 

Willing  to  adopt  ne 
ideas/practice 

Skeptical  and  cauti 
aboutchange 

Committed  to  existing  proce^ 


Organizational, 

Cultural 

Characteristics 


u  rages  change 


Values,  encourages 
innovation 


Encou  rages  flexibility  to  deal 
with  unusual  circumstances 


swards  consistency  with  existing 
cesses 


Stable  requirements  and 
technologies 

ll/willing  to  repeat  history  of  failed 
>jects-  “burned  before” 

hanging  requirements  and 
emerging  technologies 

Novel  projector  urgent  need 


Misaligned 


Behavioral 

Characteristics 


Willing  to  accept 
risk,  comfortable  with 
uncertainty 

Willing  to  adopt  new 
ideas/practice 


Skeptical  and  cautious 
aboutchange 


Committed  to  existing  processes 


Organizational, 

Cultural 

Characteristics 


ou  rages  change 


Values,  encourages 
innovation 


Encou  rages  flexibility  to  deal 
with  unusual  circumstances 


wards  consistency  with  existing 
•cesses 


Stable  requirements  and 
tecl/iologies 

Ui/willing  to  repeat  h  istory  of  failed 
p/ojects-  “burned  before” 

hanging  requirements  and 
emerging  technologies 

Novel  projector  urgentneed 


4, 

* 

V 

Project 

Project 

Project 

Circumstances 

Circumstances 

!  1 _ 1 

Circumstances 

Conflicting 


Behavioral 

Characteristics 


Willing  to  accept 
risk,  comfortable  with 
uncertainty 

Willing  to  adopt  nev 
ideas/practice 


Organizational, 

Cultural 

Characteristics 


Skeptical  and  cautic 
aboutchange 


Committed  to  existing  proces 


;ou rages  change 


Values,  encourages 
innovation 


Encou  rages  flexi  bil  ity  to  deal 
with  unusual  circumstances 


rewards  consistency  with  existing 
irocesses 


Users 


Staple  requirements  and 
te/hnologies 

Inwilling  to  repeat  history  of  failed 
projects  -  “burned  before” 

Changing  requirements  and 
emerging  technologies 

Novel  projector  urgentneed 


Novel  project  that  some  believed 
couldn’t  be  done 
“Intrapraneur”  that  was  willing  to 
challenge  establishment  and 
showed  that  he  could  deliver 
Top  cover  from  senior  leadership 


Novel  project  that  some  believed 
demanded  different  process 
Senior  leadership  encouraged 
innovation  and  prepared  to  provide 
top  cover 

Project  management  committed  to 
existing  processes;  not  prepared  to 
take  advantage  of  offered  top  cover 


Project  entailed  shift  to  cross¬ 
agency  information  sharing  model 
Development  team  implemented 
innovative  approach;  strongly 
supported  by  senior  management 
Users  comfortable  with  existing 
processes  and  tools;  resistant  to 
change 
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Implications  for  Acquisition 


■  Innovative  practices  are 
strongly  supported  by 
statute  and  regulations... 


But  challenged  by  long¬ 
standing  practices  and  culture 
(we’ve  always  done  it  that  way) 


■  Innovative  practices  are 
needed  to  meet  legitimately 
changing  user  needs  and 
provide  responsive  support 
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Summary:  A  Change  in  Culture  and 
Perspective 

■  Understand  that  different  parts  of  the  program  face  different 
types  and  degrees  of  uncertainty  and  urgency;  be  prepared 
to  tailor  acquisition  strategies  accordingly. 

■  Balance  the  need  for  discipline  and  agility  -  discipline  in 
executing  the  current  increment  and  agility  in  adjusting 
subsequent  increments  to  changing  needs,  priorities  and 
evolving  technologies 

■  Recognize  that  law  and  regulation  provide  great  flexibility 
but  implementing  “non-traditional”  practices  are  often 
challenged  by  long  standing  practice  and  culture 

■  Seek  out  active  partnership  with  the  end  user  and  “top 
cover”  within  the  organization 

■  Strive  to  implement  a  culture  that  emphasizes  outcomes, 
“time  to  value”  and  responsiveness 
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Thoughts  on  Implementing 


■  Recognize  that  any  change  will 
advance  through  stages 

■  Develop  the  vision  (SECDEF 
Gates,  DSB,  others) 

■  Empower  others  to  act  on  that 
vision 

■  Focus  on  shaping  the  mind  set 
as  well  as  defining  and 
documenting  practices 

■  Provide  training  to  key  players 
(PMs,  CEs,  COs,  others) 

■  Build  out  tools  to  help  them 
work  through  the  decision  logic 
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Background:  How  Venture  Capital  Manages 
Uncertainty 


External 
Uncertainties  (in 
the  Market) 

+ 

Internal 
Uncertainties  (in 
the  Portfolio  Firm) 


Venture  Capital  Practices 


Selection 

Market  focus 

Syndication 

Due  diligence 

Diversificatior 

Portfolio  of  “many  small  bets 
vs.  few  large  ones” 

Rapid  start  and  stop 

Agility 

Funding  tied  to  achieving 
“inchstones” 

Flexible,  opportunistic  financing 

Speed  to  market  with  feedback 

Active 

involvement 

| 

Strategic  advice 
Teambuilding/recruiting 
Leveraging  contacts... 
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Regulations  Support  Tailoring  Practices 


FAR  Part  1.102:  Statement  of  Guiding  Principles  for  the  Federal 

Acquisition  System 

(d)  The  role  of  each  member  of  the  Acquisition  Team  is  to  exercise  personal  initiative 
and  sound  business  judgment  in  providing  the  best  value  product  or  service  to  meet 
the  customer’s  needs.  In  exercising  initiative,  Government  members  of  the 
Acquisition  Team  may  assume  if  a  specific  strategy,  practice,  policy  or  procedure  is 
in  the  best  interests  of  the  Government  and  is  not  addressed  in  the  FAR,  nor 
prohibited  by  law  (statute  or  case  law),  Executive  order  or  other  regulation,  that  the 
strategy,  practice,  policy  or  procedure  is  a  permissible  exercise  of  authority. 


DoD  5000.2  Section  1.  Defense  Acquisition  Management  System 

b.  Consistent  with  this  Instruction  and  Reference  (b),  the  Program  Manager  (PM)  and  the 
MDA  shall  exercise  discretion  and  prudent  business  judgment  to  structure  a  tailored, 
responsive,  and  innovative  program. 
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Governance:  Managing  a  Program  as  a 
Portfolio  of  Investments 


Board  of  Directors 

PM 

Steering  Co / 
Investment 
Board 

User 

Rep(s) 

i 

DPM 

Chief 

Engineer 

1 - 

- 1 - 

Project 

Bus 

T&E 

Mgr  1 

Mgr 

Mgr 

,1 

,i 

Project 

1 

- 

Portfolio  of 
Projects 


■  Treat  individual  projects  as 
investments 

■  Manage  investments  as  a 
portfolio 

Allocate  resources  to  individual 
projects 

-  Continuously  monitor 

■  Market  conditions  (demand, 
competitors...) 

■  Product  delivery 

■  Fit  with  user  expectations 
Rebalance  portfolio  as  warranted 

■  Turn  projects  on/off; 
accelerate/slow  down 

■  As  necessary,  fund  risk 
hedging  initiatives  (parallel 
paths) 


MITRE 


©  2009  The  MITRE  Corporation.  All  rights  reserved. 


